Information processing apparatus, method, program, and information processing system

ABSTRACT

An information processing apparatus of the present invention includes an acquisition unit that acquires space information indicative of a position of a physical object in a first space around a first user, a space construction unit that constructs, on the basis of the space information, a shared space that is shared by the first user and a second user who exists in a second space different from the first space and in which the position of the physical object in the first space is reflected, and a determination unit that determines a position of the second user in the shared space.

TECHNICAL FIELD

The present invention relates to an information processing apparatus, amethod, a program, and an information processing system.

BACKGROUND ART

In recent years, a technology has been examined by which a movement of abody and a space are shared on a real time basis by a plurality of userswho are present in remote places spaced away from each other in such amanner as to allow the users to have such an experience that they feelas if they were in the same place. For example, a technology calledTelexistence provides an environment in which an operation and so forthare performed on a real time basis while a user is allowed to feel,through a head-mounted display or the like, as if something or someperson in a remote place were present near the user.

SUMMARY Technical Problem

For the technology described above, it is demanded to provide a novelviewing experience to a user.

Taking the problem described above into consideration, it is one ofobjects of the present invention to provide a novel viewing experienceto a user.

Solution to Problem

In order to solve the problem described above, an information processingapparatus of one aspect of the present invention includes an acquisitionunit that acquires space information indicative of a position of aphysical object in a first space around a first user, a spaceconstruction unit that constructs, on the basis of the spaceinformation, a shared space that is shared by the first user and asecond user who exists in a second space different from the first spaceand in which the position of the physical object in the first space isreflected, and a determination unit that determines a position of thesecond user in the shared space.

It is to be noted that any combinations of the foregoing as well as thecomponents and representations of the present invention as they areconverted between methods, apparatuses, programs, transitory ornon-transitory storage media in which a program is stored, systems, andso forth are also effective as aspects of the present invention.

Advantageous Effect of Invention

According to the present invention, a novel viewing experience can beprovided to the user.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an overview diagram of an information processing system.

FIG. 2 depicts an example of utilization of an embodiment by a user.

FIG. 3 is a functional block diagram of the information processingsystem.

FIG. 4A exemplifies an AR space video displayed by an HMD of a user A.

FIG. 4B exemplifies another AR space video displayed by the HMD of theuser A.

FIG. 5 is a sequence diagram depicting a flow of processing in theinformation processing system.

FIG. 6A exemplifies an AR space video displayed by the HMD of the userA.

FIG. 6B exemplifies another AR space video displayed by the HMD of theuser A.

FIG. 7 exemplifies an AR space video displayed by an HMD of a user B.

FIG. 8 is a sequence diagram depicting a flow of processing in theinformation processing system.

FIG. 9 is an overview diagram of the information processing system.

FIG. 10 is a functional block diagram of the information processingsystem.

FIG. 11 is a sequence diagram depicting a flow of processing in theinformation processing system.

DESCRIPTION OF EMBODIMENTS First Embodiment

FIG. 1 is an overview diagram of an information processing system 10according to an embodiment. The information processing system 10 of FIG.1 includes a plurality of information processing terminals 100. Theplurality of information processing terminals 100 are individuallyconnected for data communication to each other through a communicationnetwork 5 such as the Internet. The information processing system 10 ofthe present embodiment includes two information processing terminals,i.e., an information processing terminal 100A used by a user A andanother information processing terminal 100B used by a user B. However,the information processing system 10 is not limited to this and mayinclude three or more information processing terminals 100.

The information processing terminals 100A and 100B each include acontrol unit 11, a storage unit 12, a communication unit 13, and aninterface unit 14. The information processing terminals 100A and 100Bare each connected to a head-mounted display (HMD) 15, a stereo camera16, a tracker 17, and an inputting device 18. The control unit 11includes a processor and executes a program stored in the storage unit12 to execute various kinds of information processing. The storage unit12 includes a memory device such as a random access memory (RAM) andstores a program to be executed by the control unit 11 and data to beprocessed by the program. The communication unit 13 is a communicationinterface for performing data communication through the communicationnetwork 5. In the present embodiment, the information processingterminals 100A and 100B communicate with each other through P2P(peer-to-peer) connection via the respective communication units 13. TheP2P connection establishes a low-latency, high picture quality, andbidirectional communication path in comparison with connection between aserver 200 hereinafter described and an information processing terminal100.

The interface unit 14 is an interface for data communication among theHMD 15, the stereo camera 16, the tracker 17, and the inputting device18. The information processing terminals 100 are each connected to theHMD 15, the stereo camera 16, the tracker 17, and the inputting device18 by wired or wireless connection via the interface unit 14. Inparticular, the interface unit 14 includes a multimedia interface suchas the high-definition multimedia interface (HDMI) (registeredtrademark) that is a standard for a communication interface fortransmission of a video and sound using digital signals. Further, theinterface unit 14 includes a data communication interface such as auniversal serial bus (USB) interface in order to receive video signalstransmitted from the stereo camera 16.

The HMD 15 is a viewing device that is mounted on the head of a user toallow the user to appreciate a still image or a moving image displayedon a display of the viewing device and listen to sound or musicoutputted from headphones. The HMD 15 has a gyro sensor and anacceleration sensor provided therein. The HMD 15 uses the sensors tomeasure positional information of the head of the user wearing the HMD15 and a rotational angle, a tilt, and so forth of the head. The HMD 15has a camera unit incorporated therein. The HMD 15 can capture an imageof the external world by using the camera unit while the HMD is worn bythe user.

The stereo camera 16 includes a plurality of imaging elements arrangedin a lined up relation with each other. By analyzing images captured bythe imaging elements, the information processing terminal 100 canmeasure a distance from an imaging position to a physical objectexisting within a range of a field of view of the stereo camera 16.Consequently, the information processing terminal 100 can estimate aposition and a shape of a physical object existing in a space around thestereo camera 16. In the present embodiment, the stereo camera 16includes, in the range of the field of view thereof, the user andphysical objects such as a floor and a wall existing around the user andis used to specify positions of them.

The tracker 17 includes sensors such as an inertial sensor, ageomagnetic sensor, an acceleration sensor, and a motion sensor and candetect a position and a posture of a body part of the user. Theinputting device 18 includes a plurality of inputting devices such as aplurality of operation buttons, an analog stick capable of inputting ananalog amount, and so forth. The inputting device 18 supplies operationinformation inputted by the user through the inputting devices to thecontrol unit. The inputting device 18 includes a plurality of inputtingdevices such as a plurality of push-type operation buttons, an analogstick capable of inputting an analog amount, and a rotational button.

By estimating the posture and the position of the whole body of theusers A and B on the basis of detection data of the HMD 15, the stereocamera 16, and the tracker 17, posture information of the users A and Bis generated individually. The posture information of the users A and Bis reflected on a shared space hereinafter described which is shared bythe users A and B.

FIG. 2 depicts an example of utilization of the present embodiment byusers. As depicted in FIG. 2 , the users A and B exist in spacesdifferent from each other in the real world. In particular, the user Aexists in a first space 1, and the user B exists in a second space 2different from the first space 1. The HMDs 15A and 15B are mounted onthe heads of the users A and B, respectively. The first space 1 and thesecond space 2 in the present embodiment are rooms of the users A and B,respectively.

In the present embodiment, the HMD 15A of the user A mainly functions asa video see-through type HMD capable of capturing a video of theexternal world with use of the camera unit incorporated therein anddisplaying the video on a display panel. The video see-through type HMDcan generate and display a video of augmented reality (AR) (AR spacevideo) by superimposing a virtual object generated by computer graphics(CG) on the video of the external world. It is to be noted that, in acase where an avatar of the user A falls from a floor face of the sharedspace, for example, the HMD 15A in the present embodiment can functionas a non-see-through type HMD capable of displaying a video of virtualreality (VR) (VR space video) separated from the real world ashereinafter described.

In the present embodiment, the HMD 15B of the user B is anon-see-through type HMD capable of displaying a VR space video on adisplay panel thereof. The non-see-through type HMD displays, if theuser B turns the head, a VR space video of an overall circumference over360 degrees.

The stereo cameras 16A and 16B are disposed in front of the respectiveusers A and B. The stereo cameras 16A and 16B include, in the ranges oftheir fields of view, the users A and B, floor faces on which the usersA and B stand, wall faces of the rooms in which the users A and B arepresent, and so forth, respectively.

The tracker 17A is mounted on both hands of the user A, and the tracker17B is mounted on both hands and both feet of the user B. While anexample in which the trackers are mounted on both hands and both feet ofthe users is described here, the trackers 17 may be mounted only on thehands or the feet, or otherwise the trackers 17 may be mounted on otherbody parts such as the trunk. The inputting devices 18A and 18B are heldby both hands of the respective users A and B.

In the present embodiment, two spaces, i.e., the first space 1 and thesecond space 2 that is at a place spaced from the first space 1, arecoupled by a shared space hereinafter described, to thereby provide bothof the users A and B with such an experience that they can interact asif they were present at the same place (in the first space 1 on the userA side). For example, by providing such an experience that the user Aand the user B feel as if an avatar of the user B who is not present inthe first space 1 were present in the first space 1, it is possible toallow the user A and the user B to enjoy interaction and communicationtherebetween.

On the user A side, a video see-through experience with an AR spacevideo is provided. For example, the user A can destroy a virtual objectsimulating a real wall in the AR space video by performing interactionin which the body of the user A is used. Further, in the AR space video,the avatar of the user B looks as if it were present at the place andcan perform communication using the body such as a gesture.

On the user B side, an experience with a VR space video is provided. Forexample, the user B can enter, as the body of the avatar thereof, theroom of the user A (first space 1) in the VR space video. Also the userB can similarly destroy a virtual object simulating a real wall in theVR space video by performing interaction in which the body of the user Bis used. Further, in the VR space video, the body of the avatar of theuser B can be made larger and smaller in size.

FIG. 3 is a functional block diagram of the information processingsystem 10 according to the present embodiment. The functional blocks ofthe figures including FIG. 3 can be implemented in various forms byhardware only, by software only, and or by a combination of them. Theinformation processing terminal 100A includes a first acquisition unit51, a space construction unit 52, a space updating unit 53, and a firstdrawing unit 54. The information processing terminal 100B includes asecond acquisition unit 55, a provision unit 56, and a second drawingunit 57.

The first acquisition unit 51 acquires space information indicative ofpositions, colors, and textures of physical objects in the first space 1around the user A. The “physical objects” here include not only thefloor, walls, a ceiling, and articles (table and so forth) in the spacein which the user is present but also the body itself of the user. Thefirst acquisition unit 51 is an example of an acquisition unit. Thesecond acquisition unit 55 acquires shared space information and spaceupdate information hereinafter described.

The first acquisition unit 51 and the second acquisition unit 55 acquireposture information indicative of the posture of the respective users Aand B. As described hereinabove, the posture information is generated onthe basis of the detection data of the HMD 15, the stereo camera 16, andthe tracker 17. The first acquisition unit 51 and the second acquisitionunit 55 acquire operation information indicative of a desired operationof the respective users in response to an operation input made by therespective users A and B. The operation information is generated on thebasis of an operation input made through the inputting device 18.

The space construction unit 52 constructs a shared space on which thepositions of physical objects in the first space 1 are reflected on thebasis of the space information. The space construction unit 52 generatesshared space information indicative of a position coordinate system(hereinafter referred to as a shared coordinate system) of theconstructed shared space.

The space updating unit 53 updates the shared space on the basis of thespace information as well as the posture information and the operationinformation of the users A and B. As a result, the position for the userB in the shared space is determined. The space updating unit 53generates space update information indicative of a state of the updatedshared space. The space updating unit 53 is an example of adetermination unit.

The provision unit 56 provides the posture information and the operationinformation of the user B acquired by the second acquisition unit 55 tothe first acquisition unit 51. The first drawing unit 54 and the seconddrawing unit 57 each draw an appearance in the shared space with use ofa video generated using the space update information and cause theappearance to be displayed on the HMD 15. At this time, the firstdrawing unit 54 and the second drawing unit 57 specify, on the sharedcoordinate system, the position of the head of the respective users Aand B who view the shared space image thus displayed and dispose virtualcameras at the specified positions. Then, the first drawing unit 54 andthe second drawing unit 57 draw an appearance obtained when the insideof the shared space is viewed from the disposed virtual cameras.Consequently, each of the users A and B can view the appearance in theshared space together with the other user who is present at a remoteplace.

FIGS. 4A and 4B exemplify AR space videos displayed by the HMD 15A ofthe user A. In the example of FIG. 4A, a virtual object of a sword issuperimposed at a hand of the user A in the AR space video. Further, inthe AR space video, a virtual object of a humanoid avatar of the user Bis superimposed at a middle portion of the room. Therefore, displayedthrough the AR space video on the HMD 15A of the user A is an appearancein which the user A is fighting against the humanoid avatar of the userB displayed at the middle portion of the drawing, with the sword held bythe user A on the near side in the drawing, in the room in which theuser A is present (first space 1). In the example of FIG. 4B, the user Ais being attacked by the avatar of the user B.

A flow of processing by the information processing terminals 100A and100B according to the present embodiment is described with reference toa sequence diagram of FIG. 5 . In the present embodiment, theinformation processing terminal 100A of the user A functions as aserver.

First, in S11, the information processing terminal 100A establishesconnection to the information processing terminal 100B. In the presentembodiment, the information processing terminal 100A connects to theinformation processing terminal 100B by P2P connection through thecommunication network 5. In the present embodiment, the informationprocessing terminal 100A issues a notification of game participationacceptance, approves a game play request from the information processingterminal 100B, and then establishes P2P connection to the informationprocessing terminal 100B.

In S12, the first acquisition unit 51 acquires space informationindicative of the position, color, and texture of a physical object inthe first space 1 around the user A. In the present embodiment, thefirst acquisition unit 51 analyzes an image captured by the stereocamera 16A to specify the position, color, and texture of the physicalobject existing in an imaging range of the stereo camera 16A. The firstacquisition unit 51 acquires the space information on the basis of thespecified position, color, and texture of the physical object.

In particular, the first acquisition unit 51 calculates, for each of aplurality of unit regions obtained by dividing the imaging range of thestereo camera 16 in a grid pattern, the distance to a portion of animaging target captured in the unit region. By this, the firstacquisition unit 51 can acquire a distance image (depth map) includinginformation of the distance to the physical object captured in each unitregion. In the following description, a portion of a physical objectexisting in the real space, which portion is captured in a unit regionin the distance image, is referred to as a unit portion.

The first acquisition unit 51 uses numerical values of the position ofeach unit region in the distance image (that is, a direction of the unitregion as viewed from an installation position of the stereo camera 16)and the distance to the unit portion captured in the unit region tocalculate position coordinates of the unit portion in the real space.The position coordinates are three-dimensional position coordinates on acoordinate system in which the installation position of the stereocamera 16 is a reference position. By using the individual positioncoordinates of the plurality of unit portions obtained in this manner,the position and shape of a physical object having a complicatedstructure such as the body of the user or a table are specified.Further, for the position coordinates of each of the plurality of unitportions obtained in this manner, the color and the texture arespecified on the basis of the captured image.

The space information includes floor face information that designatesthe floor face in the first space 1. In the present embodiment, thefirst acquisition unit 51 generates the floor face information bydetermining a flat plane configured from a unit portion existing at thelowest position as the floor face.

In S13, the space construction unit 52 constructs a shared space on thebasis of the space information. In S13, the space construction unit 52constructs a shared coordinate system on the basis of the positioncoordinates specified in the space information, in such a manner thatthe position of the physical object in the first space 1 is reflected.Further, in the shared coordinate system, a floor face of the sharedspace is set on the basis of the floor face information included in thespace information, in such a manner that the floor face of the sharedspace corresponds to the floor face of the first space 1. It is to benoted that, in a space outside the space corresponding to the firstspace 1 in the shared space, a predetermined virtual object (forexample, such a virtual object that represents the cosmic space) isdisposed. In the present embodiment, an initial position of the user Bis set to a predetermined position in the space corresponding to thefirst space 1 in the shared space.

In S14, the space construction unit 52 generates shared spaceinformation indicative of the shared coordinate system of theconstructed shared space and provides the generated shared spaceinformation to the information processing terminal 100B.

In S15, the second drawing unit 57 generates a VR space video on thebasis of the shared space information, in such a manner that theconstructed shared space is reflected. The second drawing unit 57generates and draws the VR space video in such a manner that, on thebasis of the shared coordinate system, a floor face of the second space2 is adjusted to coincide with the floor face set in the shared spaceand that the walls, the ceiling, and the articles are formed withreference to the floor face. By displaying the VR space video in thismanner, the user B can confirm the state of the user B in the sharedspace with use of the VR space video. The user B can execute anoperation input on the basis of the VR space video in which only theposition of the physical object is reflected.

In S16, the first acquisition unit 51 acquires the space information ofthe first space 1 and the posture information and the operationinformation of the user A and acquires the posture information and theoperation information of the user B provided thereto via the provisionunit 56 of the information processing terminal 100B. Here, the operationinformation can include motion information for causing, for example, theavatar of a user to perform a specific motion (for example, shooting abullet, drawing a sword, or the like). Further, in a case where an HMDfor a VR space video like the HMD 15B of the user B is used, theoperation information further includes movement information for causingthe avatar of the user to move to a desired position in the shared spaceand scale information for setting a scale for the avatar of the user.The scale information includes a scale value for adjusting the scale ofthe avatar of the user.

In S17, the space updating unit 53 updates the shared space on the basisof the space information of the first space 1 and the postureinformation and the operation information of the users A and B acquiredin S16. In particular, the space updating unit 53 determines, for eachunit portion, the position, shape (including size), color, and textureof the virtual object in the shared space. The virtual object in thepresent embodiment is, for example, the floor, the walls, the ceiling,the articles, the avatars of the users A and B, the sword, the bullet,or the like. It is to be noted that each of the virtual objects of thefloor, the walls, the ceiling, and the articles in the shared space isconfigured from an aggregate of fragments, so that it can be destroyed.

The space updating unit 53 determines the position and the posture ofthe user A in the shared space on the basis of the space information andthe posture information of the user A in such a manner that the positionand the posture of the user A in the shared space correspond to theposition and the posture at present of the user A in the first space 1.The space updating unit 53 updates the position and the posture of thevirtual object of the avatar of the user A in the shared space on thebasis of the determined position and posture of the user A.

The space updating unit 53 determines the posture of the user B in theshared space on the basis of the posture information of the user B insuch a manner that the posture of the user B in the shared spacecorresponds to the posture at present of the user B, and determines theposition of the user B in the shared space to the position designated bythe movement information. Further, in the present embodiment, the spaceupdating unit 53 determines the size of the virtual object of the avatarof the user B in the shared space on the basis of the scale informationof the user B. For example, in a case where the designated scale isdifferent from the scale at present on the basis of the scale value ofthe scale information, the space updating unit 53 changes the size toincrease or decrease the scale of the avatar of the user B in the sharedspace. The space updating unit 53 updates the position, posture, andscale of the virtual object of the avatar of the user B in the sharedspace on the basis of the determined position, posture, and scale of theuser B.

The space updating unit 53 determines the position, color, and textureof each of the floor, the walls, the ceiling, and the articles in theshared space on the basis of the space information in such a manner thatthe position, color, and texture of each of the floor, the walls, theceiling, and the articles in the first space 1 at present are reflected.The space updating unit 53 updates the position, color, and texture ofeach of the virtual objects of the floor, the walls, the ceiling, andthe articles in the shared space on the basis of the determinedposition, color, and texture of the floor, the walls, the ceiling, andthe articles.

In the present embodiment, the space updating unit 53 determines, on thebasis of the space information and the posture information and theoperation information of the users A and B, whether or not a virtualobject relating to the avatar of at least one of the users A and B isbrought into contact with a first space virtual object that forms thespace corresponding to the first space 1 in the shared space. Here, thevirtual object relating to the avatar in the present embodiment is avirtual object that is operated by the avatar such as a predeterminedbody part (arm or the like) of the avatar, a sword held by the avatar,or a bullet shot by the avatar. Further, the first space virtual objectin the present embodiment is a virtual object that forms the floor,walls, ceiling, and articles corresponding to the first space 1 in theshared space. However, this is not restrictive, and the first spacevirtual object may be a virtual object that forms at least one of thefloor, walls, ceiling, and articles corresponding to the first space 1in the shared space. The first space virtual object includes a pluralityof virtual objects that are fragments of the same.

In a case where it is determined that the virtual object relating to theavatar is brought into contact with the first space virtual object, thespace updating unit 53 changes the state of the contact portion of thefirst space virtual object. This change of the state includes, forexample, destruction, bursting into flames, freezing, or coloring of thecontact portion of the first space virtual object. In a case where thechange of the state is, for example, destruction of the contact portionof the first space virtual object, the space updating unit 53 changesthe state of the contact portion of the virtual object in such a mannerthat the contact portion of the virtual object disappears in the sharedspace.

In the present embodiment, in a case where the first space virtualobject corresponding to the floor face of the shared space is destroyedand at least one of the avatars of the users A and B is positioned atthe destroyed portion of the floor face, the space updating unit 53determines the position of the user relating to the avatar positioned atthe destroyed portion of the floor face in the shared space in such amanner that the avatar falls from the position of the destroyed floorface.

Further, in a case where it is determined that, for example, a virtualobject of a sword held by the avatar of one of the users is brought intocontact with a virtual object of a bullet shot from the other user, thespace updating unit 53 generates a virtual object in such a manner as toperform a specific motion such as bouncing back the bullet. As a resultof such interactions performed among the users A and B and the virtualobjects in the shared space as described above, the virtual objects areincreased or decreased or changed in appearance in S19A and S19Bhereinafter described.

In S18, the space updating unit 53 provides the generated space updateinformation to the first drawing unit 54 and also to the informationprocessing terminal 100B.

In S19A, the first drawing unit 54 draws an AR space video on the basisof the space update information in such a manner that the updated sharedspace is reflected. In S19A, the first drawing unit 54 draws the virtualobjects obtained when the updated shared space is viewed from aviewpoint position and a view direction of the user A wearing the HMD15A. In the present embodiment, the first drawing unit 54 lays down,according to the positions of the floor, walls, ceiling, and articles inthe first space 1 within the field of view of the HMD 15A, the virtualobjects of the fragments of them, to thereby draw a first space virtualobject in such a manner that the floor, walls, ceiling, and articles ofthe first space 1 are reflected in the shared space. The first drawingunit 54 superimposes, by a post process, a captured image (videosee-through image) of the floor, walls, ceiling, and articles capturedby the HMD 15 on a surface of the generated first space virtual objectof the floor, the walls, the ceiling, and the articles. Here, the“surface of the first space virtual object” is a face of the first spacevirtual object obtained when it is viewed from within the spacecorresponding to the first space 1 in the shared space. The firstdrawing unit 54 draws an AR space video by superimposing the virtualobjects of the avatar of the user B, a sword, and so forth on thecaptured image superimposed on the virtual objects, on the basis of thespace update information.

In a case where the first space virtual object (wall or the like) isdestroyed, the virtual objects of the fragments laid down at thedestroyed portion of the virtual object are broken into pieces anddisappear. As a result, it becomes possible for the first drawing unit54 to draw, for the disappearing portion, a virtual object existing on afar side in the view direction of the user A. Further, a texture imageindicating a cross section obtained when a wall or the like is destroyedis pasted to a destruction cross section of the virtual objects of thefragments which have been broken into pieces. In addition, as describedhereinabove, on the surface of the first space virtual object, thecaptured image (video see-through image) of the HMD 15 is superimposedby the post process. As a result, if the first space virtual object of awall or the like is destroyed, then the image indicated by the virtualobjects of the fragments changes from the video see-through image to apredetermined texture image that indicates the destroyed state. Thisenables such presentation that the wall or the like at the contactportion looks as if it were destroyed actually. In this case, for thevideo see-through image, a texture image generated from the capturedimage captured by the HMD 15 may be used or otherwise the captured imagemay be used as it is.

In a case where the avatar of the user A is positioned at the destroyedportion of the floor face in the shared space, the position of the userA in the shared space is determined in such a manner that the avatar ofthe user A falls from the position of the destroyed floor face. In thiscase, the HMD 15A of the user A displays a VR space video in which aspace outside the space corresponding to the first space 1 in the sharedspace is reflected, due to the falling of the avatar of the user A fromthe space corresponding to the first space 1 in the shared space.

In S19B, the second drawing unit 57 draws a VR space video on the basisof the space update information, in such a manner that the updatedshared space is reflected. In S19B, the second drawing unit 57 draws aVR space video by generating, on the basis of the space updateinformation, virtual objects obtained when the updated shared space isviewed from a viewpoint position and a view direction of the user Bwearing the HMD 15B. In particular, the second drawing unit 57 draws aVR space video by generating virtual objects in such a manner that thefloor, the walls, the ceiling, the articles, the avatar of the user A,the sword, and so forth are reflected in the shared space. For example,the second drawing unit 57 draws a VR space video in such a manner thatthe surface of the first space virtual object indicates an image inwhich the color and texture of the floor, walls, ceiling, and articlesof the first space 1 are reflected. In a case where the virtual objectsof the fragments of the contact portion disappear as described above,the second drawing unit 57 draws, for the disappearing portion, avirtual object existing on the far side in the view direction of theuser B.

If the first space virtual object is destroyed, then an image indicatedby the virtual objects of the fragments changes from the image in whichthe color and texture of the destroyed portion are reflected to atexture image indicative of the destroyed state. As a result, suchpresentation as if the wall or the like at the contact portion wereactually destroyed becomes possible. It is to be noted that S19A andS19B are executed at the same time.

Thereafter, the information processing system 10 repeatedly executes theprocesses from S16 to S19A and S19B. In particular, the informationprocessing system 10 repeatedly executes acquisition of spaceinformation of the first space 1, acquisition of posture information andoperation information of each of the users A and B, and updating anddrawing of the shared space according to the acquired pieces ofinformation. By this, an appearance of the shared space in which theavatars reflecting a movement of the body and so forth of the users Aand B exist can be presented to the users A and B.

The processing of the information processing terminals 100A and 100B iscompleted accordingly.

According to the present embodiment, it becomes possible for each of theusers A and B to interact with the user present in the other space, withuse of a shared space in which physical objects in the room of the userA are reflected. Therefore, a novel experience can be provided to theusers.

In the present embodiment, the position of the user B is determined in aspace corresponding to the first space 1 in the shared space. As aresult, to the user A, an AR space video is drawn in such a manner thatthe avatar of the user B appears in the room of the user A itself, andto the user B, a VR space video is drawn as if the user B were presentin the room of the user A. Therefore, it becomes possible to provide anexperience with more immersive and realistic feelings to the users A andB.

In the present embodiment, the space updating unit 53 determines a scalefor an avatar of a user in the shared space on the basis of scaleinformation. According to the present configuration, it becomes possibleto change the scale of the avatar of each user in the shared space.Therefore, since the users A and B can interact, with the avatarsdifferent in scale from each other in the shared scale, it becomespossible to provide a more novel experience.

In the present embodiment, the space construction unit 52 sets the floorface of the shared space in such a manner that the floor face of theshared space corresponds to the floor face of the first space 1. Sincethis causes the floor face of the user A and the floor face of the userB to coincide with each other, the users A and B can have such a feelingthat they can perform interaction in the same room.

In the present embodiment, the space information indicates the color andtexture of a physical object in addition to the position of the physicalobject. According to this configuration, since a video closer to theactual room of the user A (first space 1) is obtained, it becomespossible to provide an experience with a more realistic feeling.

In the present embodiment, in a case where a virtual object relating tothe avatar of at least one of the users A and B is brought into contactwith a first space virtual object, the state of the contact portion ofthe first space virtual object is changed. According to the presentconfiguration, since specific presentation is performed for a wall orthe like of the actual room of the user A, it becomes possible toprovide an experience with a more realistic feeling.

In the present embodiment, in a case where a first space virtual objectcorresponding to the floor face of the shared space is destroyed and atleast one of the avatars of the user A and the user B is positioned atthe destroyed portion of the floor face, the position of the userrelating to the avatar in the shared space is determined in such amanner that the avatar positioned at the destroyed portion of the floorface falls from the position of the destroyed floor face. According tothe present configuration, it becomes possible to provide, to the user,such a novel experience that the user feels as if the user fell in errorfrom the actual room of the user A.

In the present embodiment, the surface of the first space virtual objectindicates an image indicative of physical objects in the first space 1(for example, a video see-through image or an image in which the colorand texture of the physical objects are reflected), and the destructioncross section of the first space virtual object indicates apredetermined texture image indicative of the destroyed state. Accordingto the present configuration, since such presentation as if the room ofthe user A were destroyed actually becomes possible, it is possible toprovide a more novel experience.

In the following, modifications are described. While, in the presentembodiment, the HMD 15A functions principally as a video see-throughtype HMD and the HMD 15B functions as a non-see-through type HMD, thisis not restrictive. Each of the HMDs 15A and 15B may function as eithera video see-through type HMD or a non-see-through type HMD.

While, in the present embodiment, the position of the floor of the firstspace 1 on the user A side and the position of the floor of the secondspace 2 on the user B side are made to coincide with each other, this isnot restrictive. For example, any one of the floor, a wall, and theceiling on one user side may be made to coincide with the floor, a wall,or the ceiling on the other user side.

While, in the present embodiment, the information processing terminal100A functions as a server, this is not restrictive. For example, theinformation processing terminal 100B may function as a server. While, inthe present embodiment, the space information indicates the position,color, and texture of a physical object in the first space 1, this isnot restrictive. It is sufficient if the space information indicates atleast the position of a physical object in the first space 1. While, inthe present embodiment, the space information indicates the position,color, and texture of a physical object, this is not restrictive. It issufficient if the space information indicates at least the position of aphysical object. While, in the present embodiment, operation informationis inputted through the inputting device 18, this is not restrictive.Operation information may be inputted otherwise in response to anymotion of a user such as a gesture.

Second Embodiment

In the following, a second embodiment of the present invention isdescribed. In the drawings and the description of the second embodiment,components and members identical or equivalent to those of the firstembodiment are denoted by identical reference signs. Descriptionoverlapping that of the first embodiment is omitted suitably, and aconfiguration different from that of the first embodiment is describedintensively.

In the first embodiment described above, a shared space is constructedin such a manner that the avatar of the user B appears in a spacecorresponding to the first space 1 in the shared space. In contrast, inthe second embodiment, a shared space is constructed in such a mannerthat the avatar of the user B appears outside the space corresponding tothe first space 1 in the shared space. In particular, a shared space isconstructed in such a manner that the first space 1 of the user Aappears in the second space 2 of the user B.

FIGS. 6A and 6B exemplify AR space videos displayed by the HMD 15A ofthe user A. In the example of FIG. 6A, the user B is represented not asa humanoid avatar but as an avatar of a huge robot. In this example, therobot avatar of the user B is scaled in such a manner that it has aheight of approximately 10 m in the AR space video. Further, part of awall of the room of the user A is indicated by a virtual object having amode in which it seems to have been destroyed. As a result, on the HMDof the user A, an appearance is displayed in which, when an arm of therobot of the user B is brought into contact with the wall of the room ofthe user A from the outside of the room, the robot arm destroys the wallof the room and advances into the room of the user A. In the example ofFIG. 6B, an appearance in which most part of the room of the user A hasbeen destroyed by the robot avatar of the user B is represented.

In the present embodiment, the HMD 15B of the user B is a videosee-through type HMD that can superimpose a virtual object on a video ofthe external world to generate and display an AR space video. FIG. 7exemplifies an AR space video displayed by the HMD 15B of the user B. Inthe example of FIG. 7 , both hands of the user B are represented withvirtual objects of robot arms superimposed thereon on the near side inthe drawing in the AR space video. Further, the wall of a meshindicating the room of the user A (first space) is partly destroyed, andthe avatar of the user A can be seen through the destroyed portion.Since the scale of the avatar of the user B is great, the room of theuser A is reflected to be small in the AR space video.

According to the present embodiment, the user A can perform interaction,in the room of the user A, with the avatar of the user B coming in fromthe outside of the room, and the user B can perform interaction, in theroom of the user B, with the avatar of the user A that is in the room ofthe user A. Therefore, it becomes possible to provide an experience witha more realistic feeling. In the following, processing in the presentembodiment is described with reference to FIG. 8 .

In S22, the first acquisition unit 51 acquires space information of thefirst space 1 and further acquires space information of the second space2 which information indicates the position of physical objects in thesecond space 2. The space information of the second space 2 is providedvia the provision unit 56 of the information processing terminal 100B.In the present embodiment, the second acquisition unit 55 of theinformation processing terminal 100B acquires the space information ofthe second space 2 with use of the stereo camera 16B by a techniquesimilar to that for the space information of the first space 1. Theprovision unit 56 provides the acquired space information of the secondspace 2 to the information processing terminal 100A. The spaceinformation of the second space 2 includes floor face informationindicative of the floor face of the second space 2.

In S23, the space construction unit 52 constructs a shared space on thebasis of the space information of the first space 1 and the spaceinformation of the second space 2. In S23, the space construction unit52 constructs a shared space on the basis of the position coordinates ofthe first space 1 and the second space 2 specified in the respectivepieces of space information, in such a manner that a space correspondingto the first space 1 is located at a predetermined position in a spacecorresponding to the second space 2 in the shared space. In the presentembodiment, the space corresponding to the second space 2 in the sharedspace has its scale set in such a manner that the space can accommodatethe space corresponding to the first space 1. In the space correspondingto the first space 1, the position of physical objects in the firstspace 1 is reflected. The shared coordinate system in the presentembodiment is set on the basis of the floor face information included inthe respective pieces of space information, in such a manner that thefloor face of the space corresponding to the first space 1 and the floorface of the space corresponding to the second space become horizontal toeach other. In the present embodiment, the initial position of the userB is set to a predetermined position outside the space corresponding tothe first space 1 in the shared space.

In S25 after S24, the second drawing unit 57 generates an AR space videoon the basis of the shared space information in such a manner that theshared space is reflected. In this AR space video, a box-shaped virtualobject representing the first space is superimposed at a predeterminedposition in the second space 2. It is to be noted that the position ofthe box-shaped virtual object representing the first space may bechanged on the basis of operation information of the second user (forexample, to a position on a table or the like). In S29B after S26 toS28, the second drawing unit 57 generates an AR space video on the basisof the space update information in such a manner that the updated sharedspace is reflected.

Thereafter, the information processing system 10 repeatedly executes theprocesses from S26 to S29A and S29B. In particular, the informationprocessing system 10 repeatedly executes acquisition of spaceinformation of the first space 1, acquisition of posture information andoperation information of each of the users A and B, and updating anddrawing of the shared space according to the acquired pieces ofinformation.

The processing of the information processing terminals 100A and 100B iscompleted accordingly.

Third Embodiment

In the following, a third embodiment of the present invention isdescribed. In the drawings and the description of the third embodiment,components and members identical or equivalent to those of the firstembodiment are denoted by identical reference signs. Descriptionoverlapping that of the first embodiment is omitted suitably, and aconfiguration different from that of the first embodiment is describedintensively.

The information processing system 10 of FIG. 9 includes a server 200 anda plurality of information processing terminals 100. The server 200 isconnected for data communication to the plurality of informationprocessing terminals 100 through a communication network 5. The server200 includes a control unit 211, a storage unit 212, and a communicationunit 213.

The control unit 211 includes a processor and executes a program storedin the storage unit 212 to execute various kinds of informationprocessing. The storage unit 212 includes a memory device such as a RAMand stores a program to be executed by the control unit 211 and data tobe processed by the program. The communication unit 213 is acommunication interface for performing data communication through thecommunication network 5. In the present embodiment, each of theinformation processing terminals 100 and the server 200 communicate witheach other through client-server type connection via the communicationunits 13 and 213.

FIG. 10 is a functional block diagram of the information processingsystem 10 according to the present embodiment. As depicted in FIG. 10 ,the server 200 includes a first acquisition unit 51, a spaceconstruction unit 52, and a space updating unit 53. The informationprocessing terminals 100 each include a second acquisition unit 55, aprovision unit 56, and a drawing unit 58. It is to be noted that thedrawing unit 58 has a function similar to that of the first drawing unit54 and the second drawing unit 57. The first acquisition unit 51 and thesecond acquisition unit 55 in the present embodiment are examples of aserver side acquisition unit and a terminal side acquisition unit,respectively.

A flow of processing by the information processing terminals 100 and theserver 200 according to the present embodiment is described withreference to a sequence diagram of FIG. 11 . In S31, the server 200establishes connection to each of the information processing terminals100. In the present embodiment, the server 200 establishes client-servertype connection to each of the information processing terminals 100through the communication network 5.

In S32, the first acquisition unit 51 acquires space information fromthe information processing terminals 100. In the present embodiment, thefirst acquisition unit 51 acquires space information from at least oneof the information processing terminals 100A and 100B via the provisionunit 56.

In S33, the space construction unit 52 constructs a shared space on thebasis of the acquired space information. In the present embodiment, theshared space is constructed using the space information of theinformation processing terminal 100 (in the present embodiment, referredto as the information processing terminal A) designated in advance. InS37 after S34 to S36, the space updating unit 53 updates the sharedspace on the basis of the space information of the first space 1 in theinformation processing terminal 100A and posture information andoperation information of the users A and B, which pieces of informationhave been acquired in S36. In S38, the space updating unit 53 providesspace update information to the information processing terminals 100Aand 100B. In S39, the drawing units 58 of the information processingterminals 100A and 100B individually draw an AR space video or a VRspace video on the basis of the space update information acquired viathe second acquisition unit 55, in such a manner that the updated sharedspace is reflected.

Thereafter, the information processing system 10 repeatedly executes theprocesses from S36 to S39. The processing of the information processingterminals 100 and the server 200 is completed accordingly.

The present invention has been described on the basis of theembodiments. The embodiments are exemplary, and it will be recognized bythose skilled in the art that various modifications can be made in thecombination of the components and the processes in the embodiments andthat also such modifications fall within the scope of the presentinvention.

INDUSTRIAL APPLICABILITY

The present invention relates to an information processing apparatus, amethod, a program, and an information processing system.

REFERENCE SIGNS LIST

-   1: First space-   2: Second space-   5: Communication network-   10: Information processing system-   11: Control unit-   12: Storage unit-   13: Communication unit-   14: Interface unit-   15: HMD-   16: Stereo camera-   17: Tracker-   18: Inputting device-   51: First acquisition unit-   52: Space construction unit-   53: Space updating unit-   54: First drawing unit-   55: Second acquisition unit-   56: Provision unit-   57: Second drawing unit-   58: Drawing unit-   100: Information processing terminal-   200: Server

1. An information processing apparatus comprising: an acquisition unitthat acquires space information indicative of a position of a physicalobject in a first space around a first user; a space construction unitthat constructs, on a basis of the space information, a shared spacethat is shared by the first user and a second user who exists in asecond space different from the first space and in which the position ofthe physical object in the first space is reflected; and a determinationunit that determines a position of the second user in the shared space.2. The information processing apparatus according to claim 1, whereinthe determination unit determines the position of the second user withina space corresponding to the first space in the shared space.
 3. Theinformation processing apparatus according to claim 1, wherein theacquisition unit further acquires different space information indicativeof a position of a physical object in the second space, and the spaceconstruction unit constructs the shared space further on a basis of thedifferent space information, in such a manner that one of a spacecorresponding to the first space and another space corresponding to thesecond space in the shared space is positioned in the other one of thespaces.
 4. The information processing apparatus according to claim 1,wherein the acquisition unit acquires scale information for setting ascale for an avatar of at least one of the first user and the seconduser in the shared space, and the determination unit determines thescale for the avatar on a basis of the scale information.
 5. Theinformation processing apparatus according to claim 1, wherein the spaceinformation includes floor face information that designates a floor facein the first space, and the space construction unit sets a floor face ofthe shared space on a basis of the floor face information in such amanner as to correspond to the floor face of the first space.
 6. Theinformation processing apparatus according to claim 1, wherein the spaceinformation further indicates a color and a texture of the physicalobject in the first space, and the determination unit determines a colorand a texture of a virtual object of a physical object in the sharedspace, in such a manner that the color and texture of the physicalobject in the first space are further reflected.
 7. The informationprocessing apparatus according to claim 1, wherein, in a case where avirtual object relating to an avatar of at least one of the first userand the second user is brought into contact with a first space virtualobject that forms a space corresponding to the first space in the sharedspace, a state of the contact portion of the first space virtual objectis changed.
 8. The information processing apparatus according to claim7, wherein the change of the state is destruction of the contact portionof the first space virtual object, and, in a case where the first spacevirtual object corresponding to a floor face of the shared space isdestroyed and where at least one of the avatars of the first user andthe second user is positioned at the destroyed portion of the floorface, the determination unit determines a position of the user relatingto the avatar positioned at the destroyed portion of the floor face inthe shared space in such a manner that the avatar falls from theposition of the destroyed floor face.
 9. The information processingapparatus according to claim 7, wherein the change of the state isdestruction of the contact portion of the first space virtual object,the first space virtual object includes a plurality of virtual objectsthat are fragments thereof, a surface of the first space virtual objectindicates an image that indicates the physical object in the firstspace, and a destruction cross section of the first space virtual objectindicates a predetermined texture image indicating a destroyed state.10. A method comprising: acquiring space information indicative of aposition of a physical object in a first space around a first user;constructing, on a basis of the space information, a shared space thatis shared by the first user and a second user who exists in a secondspace different from the first space and in which the position of thephysical object in the first space is reflected; and determining aposition of the second user in the shared space.
 11. A non-transitory,computer readable storage medium containing a computer program, whichwhen executed by a computer, causes the computer to perform a method,comprising: acquiring space information indicative of a position of aphysical object in a first space around a first user; constructing, on abasis of the space information, a shared space that is shared by thefirst user and a second user who exists in a second space different fromthe first space and in which the position of the physical object in thefirst space is reflected; and determining a position of the second userin the shared space.
 12. An information processing system comprising: aserver; and a plurality of information processing terminals; theplurality of information processing terminals including a firstinformation processing terminal, the first information processingterminal including a terminal side acquisition unit that acquires spaceinformation indicative of a position of a physical object in a firstspace around a first user who uses the first information processingterminal, and a provision unit that provides the acquired spaceinformation to the server; and the server including a server sideacquisition unit that acquires the space information from the firstinformation processing terminal, a space construction unit thatconstructs, on a basis of the space information, a shared space that isshared by the first user and a second user who exists in a second spacedifferent from the first space and in which the position of the physicalobject in the first space is reflected, and a determination unit thatdetermines a position of the second user in the shared space.